Crossbar switches are well known for providing arbitrary numbers of interconnections between input nodes and output nodes. Clos networks are often used for switching when the physical circuit switching needs exceeds the capacity of any reasonably feasible single crossbar switch. Clos networks employ stages of crossbar switches to interconnect the input nodes to the output nodes. An advantage of Clos networks is that the network can scale up in node count using crossbar switches of a fixed size, which is not possible in a single switch.
Clos network topology (sometimes also known as a ‘Fat Tree’ topology) is often used in high performance computing clusters. The Clos structure provides constant bisectional bandwidth (the bandwidth cut by a line drawn through the middle of the network) as node count increases, and also provides constant latency for expanding numbers of nodes in the cluster.
As typically used in interconnecting servers in a cluster computing system, Clos networks have what are known as “leaf” and “spine” stages. A leaf stage is directly coupled to a set of the input/output nodes, while a spine stage switches signals among the leaf stages, thereby enabling any leaf node to communicate with any other leaf node. In the hypothetical example of FIG. 8, one leaf stage 20 is composed of a series of line boards (1-6) having input nodes 22. Each line board includes two crossbar switches. The complete leaf stage 20 includes twelve crossbar switches—two switches per line board for each of the six boards. Leaf stage 30 is also composed of a series of line boards (7-12), each line board also including two crossbar switches. Stage 30 is connected to the output nodes 24.
The spine stage 40 is composed of four fabric boards (1′-4′) each including three crossbar switches. Each node 22 of a leaf stage 20 can be connected by the crossbar switch on the board to which it is coupled to any of the spine stage crossbar switches 1′-4′. Each crossbar switch on the spine stage 40 can be connected to the desired crossbar switch of leaf stage 30, and thereby to the desired node 24 on the right side of the illustration. In essence the structure allows every node 22 on a leaf stage 20 to be connected to any other node 24 on the other leaf stage 30. In a bidirectional system in which the nodes on the leaf stages 20 or 30 can be either input or output nodes, the signals can be routed from either leaf stage 20 to 30, or from stage 30 to 20.
Another Clos topology is known as a “folded” Clos network. In this topology the nodes coupled to stage 20 are bidirectional and only leaf stage 20 and spine stage 40 are required. (Stage 30 is eliminated.) The combination of stages 20 and 40 enable any signal arriving on any node 22 to be switched to any other node 22.
Conventional Clos topology systems use electrical crossbar switches with electrical interfaces and copper cabling. As evident from FIG. 1, an enormous number of copper cables are required to provide all of the interconnections among all of the nodes. At higher node counts, the leaf and spine switches can be combined in a proprietary chassis-based system which is connected to servers remotely through long reaches of copper cabling. The result of this configuration is that switches at the highest level of the system (spine switches) have very high node count and therefore have high cable density. The scalability of these systems is ultimately limited by the reach of the copper cabling, which at high transmission speeds is restricted.
These conventional systems can suffer from high cost of installation and management due to the complex configuration of copper cabling. The density of the system is limited by the connector density at the nodes of the leaf elements, as well as the high power required to drive high bandwidth signals through the relatively high signal loss cabling. The reliability is also reduced because the weight of the cables strains the connectors, resulting in faulty links which are difficult to isolate and replace.